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Abstract 

Have you ever sought to use metagenomic DNA sequences reported in scientific publica¬ 
tions? Were you successful? Here, we reveal that metagenomes from no fewer than 20% of 
the papers found in our literature search, published between 2016 and 2019, were not 
deposited in a repository or were simply inaccessible. The proportion of inaccessible data 
within the literature has been increasing year-on-year. Noncompliance with Open Data is 
best predicted by the scientific discipline of the journal. The number of citations, journal type 
(e.g.. Open Access or subscription journals), and publisher are not good predictors of data 
accessibility. However, many publications in high-impact factor journals do display a higher 
likelihood of accessible metagenomic data sets. Twenty-first century science demands 
compliance with the ethical standard of data sharing of metagenomes and DNA sequence 
data more broadly. Data accessibility must become one of the routine and mandatory com¬ 
ponents of manuscript submissions—a requirement that should be applicable across the 
increasing number of disciplines using metagenomics. Compliance must be ensured and 
reinforced by funders, publishers, editors, reviewers, and, ultimately, the authors. 


Drivers of open data 

Science, as an ‘institution of organised criticism’ [1], progresses through the act of building on 
communal knowledge by ‘standing on the shoulders of giants’. Information sharing has been 
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the pillar of scientific advancement ever since the first scientific journal, Philosophical Transac¬ 
tions of the Royal Society, in 1666. Open Data is the engine of science. High-throughput DNA 
sequence outputs are a relatively new form of data that offers exciting opportunities for 
hypothesis testing and retesting. Such data can best serve science if they adhere to 3 principles: 
transparency, reproducibility, and reusability [2]. GenBank of the National Center for Biotech¬ 
nology Information (NCBI), Metagenomic Rapid Annotations using Subsystems Technology 
(MG-RAST), the European Nucleotide Archive (ENA), and the DNA Data Bank of Japan 
(DDBJ) are just a few examples of online repositories of DNA sequences. Several scientific 
achievements in recent years have relied on publicly available DNA data sets (reviewed by [3]), 
including assembled genomes from uncultured organisms [4], the discovery of the CRISPR- 
Cas systems [5], and unravelling the relationship between microbiomes or genetic features and 
specific diseases [6]. 

The benefits of Open Data to science are clear. It is proposed by many governments that 
the publishing of government-funded research data in a transparent, reproducible, and reus¬ 
able format can ‘strengthen citizen engagement and yield new innovative businesses’ [7]. ‘Data 
can also be used in innovative ways that bring economic benefits to citizens and businesses by 
releasing untapped enterprise and entrepreneurship’ [8]. The leveraging of influenza virus 
(RNA) sequences [9] for the development of novel, more efficacious influenza vaccines [8] 
stands as one of a handful of clear-cut examples of the potential societal and economic benefits 
from Open Data. The emergence of the Severe Acute Respiratory Syndrome Coronavirus 2 
(SARS-CoV-2) also clearly demonstrates the synergy that can come from ensuring Open 
Access to sequencing data—bringing rapid insights that can stem the spread of a potentially 
pandemic virus in real time [10,11]. 


Current state of data availability 

We screened aU studies containing primary metagenomic data accessible through Clarivate 
Web of Science from the beginning of 2016 to March 2019 (see SI Text) to quantify the current 
state of open science in studies that generate metagenomic data. We focused on metagenomic 
data such as shotgun metagenomics because they represent a rich and rapidly growing data set 
that can be very easily and fruitfully remined and repurposed for a wide range of hypothesis 
testing. 

Of the 1,269 publications we considered, 262 (20.6%) did not provide public access, either 
by not mentioning data availability (12.9%) or by reporting nonexistent or empty accession 
numbers (7.7%; Fig lA). Conversely, 990 publications (78.0%) reported accession numbers or 
a link to the data. Five (0.4%) studies deposited sensitive data that require authorised access, 
and twelve (1%) stated that data are available from the authors upon request. Among the stud¬ 
ies that offered no public access to data, only eight represented naive mistakes, e.g., by errone¬ 
ously reporting the metagenome submission code (SUB) that NCBI provides upon 
submission, which cannot be used to retrieve the data set, or with typos in the code preventing 
data accessibility. 

Data set accessibility was significantly different across journals (generalised linear model 
[GEM], binomial: likelihood ratio chi-squared test [LR ChisqJsg = 266.9,p < 0.0001; SI Data) 
and continued to decline with each passing year (LR Chisqi = 4.9, p = 0.026). The proportion 
of unavailable published metagenomes increased (GEM, binomial: LR Chisqi = 7.5, p = 0.006; 
SI Fig) annually by 2% from 17% in 2016 (55 out of 316), 19% in 2017 (69 out of 367), and 
21% in 2018 (87 out of 414). This trend continues into 2019 (up to March; 29%, or 49 out of 
172). Accessibility of data was not correlated with the number of citations (time-regressed 
residual values SI Fig) (LR Chisqi = 3.8, p = 0.060). 
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Fig 1. Main trends in accessibility of metagenomic data. (A) Proportion of papers with accessible metagenomic data 
and type of inaccessible data (grey) (B) in B, M, or T journals. (C) Temporal trends divided by B, M, and T journals. B, 
biological; M, multidisciplinary; T, technological. 


https://dol.Org/10.1371/journal.pbio.3000698.g001 


Searching for correlates at the journal level, differences between disciplines—as defined by 
Clarivate Analytics (generalised linear mixed effects model [GLMEM], binomial: LR Chisqm = 
23.3, p = 0.0097)—were significant (S2 Fig). To better define the distribution of unavailable 
data across scientific disciplines, we assembled journals into 3 major categories (biological, 
multidisciplinary, and technological/applied sciences; see SI Text, S2 Data). Data accessibility 
was significantly better for biological journals (with 13% of unavailable metagenomic data) 
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and the multidisciplinary journals (10%) than for technological/applied sciences (43%) (Fig 
IB) (GLM, binomial: LR Chisqi = 5.9,p - 0.015; Fig 1C). We hypothesise that an increase in 
the proportion of technological/applied science papers using metagenomics can account for 
the overall temporal increase in noncompliance over time. 

Overall, the proportion of papers with available metagenomes was significantly different 
between biological, multidisciplinary, and technological/applied science journals, regardless of 
Open Access, the age of the journal, and the publisher (Table 1). Regarding publishers, none of 
the publishing companies with at least 2 journals managed to achieve full compliance with data 
accessibilities (SI Data), and the differences between publishers are only marginally significant 
(p = 0.0447; S3 Fig). This can be considered as a consequence of the huge difference in the data 
available between journals from the same publisher. For example, in the Nature Publishing 
Group, all journals share the same policies on data availability (https://www.nature.com/nature- 
research/editorial-policies/reporting-standards). Submission of metagenomic raw data to a 
community-endorsed, public repository is mandatory. Studies in the Nature Publishing Group 
journals Nature and Nature Microbiology aU complied with Open Data, while others, such as 
Scientific Reports, revealed missing data for over 35% of the articles queried in this study. 

Analysis of the impact factor reveals that the journals with the highest impact factor 
(Table 1 and S3 Fig) fuUy comply with data accessibility for metagenomes (e.g.. Nature, Nature 
Microbiology, and Science). Journals publishing higher numbers of papers per year had a lower 
proportion of accessible data (S3 Fig and Table 1), with some exceptions: Nature, Science, Peer 
J (100% compliance), and Nature Communications (97%). Six ‘high volume’ journals correctly 
provided access to raw data for only 25%-50% of their published articles containing metage¬ 
nomic data (SI Data and S4 Fig). 


Action to ensure ‘open sequencing data’ 

Noncompliance with Open Data necessitates failures by the authors, journal managers, editors, 
and reviewers. Chronic failures to ensure Open Data should also be prioritised by funders, as 
there might be insufficient disincentives for noncompliance. Differences among journal poli¬ 
cies (or lack thereof) and their ability to enforce such policies might explain some of these defi¬ 
ciencies. Technological/applied science journals are a rapidly growing publisher of studies 
containing metagenomic data and are currently the least likely publisher to comply with Open 
Data, ft is clear that the best practice between disciplines is not being shared and that 


Table 1. Potential drivers of the proportion of papers with accessible data for each journal. 


Driver 

LR Chisq 

df 

P 

Effect 

B-M-T 

16.9 

2 

0.0002 

- 

Number of papers 

10.8 

1 

0.0010 

Negative 

Impact factor 

7.1 

1 

0.0075 

Positive 

Publisher 

24.1 

14 

0.0447 

- 

Open Access 

2.6 

1 

0.1094 

- 

Age 

0.4 

1 

0.8378 

Negative 


Data are accounting for the following: the differences among biological, multidisciplinary, and technological journals 
(B-M'T), the impact factor (in log scale), the number of papers (in log scale), the publisher, Open Access, and the age 
(in log scale). Analysis of the deviance table from binomial GLM with the LR Chisq, df, p-value, and directionality of 
the effect for continuous variables. 

Abbreviations: B-M-T, biological, multidisciplinary, and technological journal; df, degrees of freedom; GLM, 
generalised linear model; LR Chisq, likelihood ratio chi-squared test 

https://doi.Org/10.1371/journal.pbio.3000698.t001 
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publishers and editors must work towards harmonising metagenomic submission require¬ 
ments across disciplines and ensure compliance is obligatory. 

On average, between 50 and 90 metagenome-containing articles per year are not available 
out of the 300 to 400 papers per year we screened, with an abrupt increase of 50 metagenomes 
out of 170 for the first months of 2019. Although the estimates of open metagenomic data 
compliance are not inclusive of ah studies containing metagenomes, we feel this study does 
represent an unbiased survey of the literature. As such, we feel that this study should be used 
as a warning of a potentiahy growing problem, particularly in journals without a tradition of 
publishing studies with metagenomes and perhaps without access to the kind of reviewer base 
that holds Open Data as a priority. 

Data avahability reflects on the transparency of the scientific research and its reliability— 
compliance is a matter of scientific integrity and should be a requirement for aU research that 
has public funding. Barriers to compliance with best practice need to be explored to ensure 
that complete Open Data is achievable. It should be noted that making data available is only 
the first step. Equally essential is the extent and accuracy to which associated metadata are pro¬ 
vided, which are needed to put results into context and inform follow-up studies. Technical 
problems might also hamper accessibility to data sets. Of the studies that reported that they 
submitted their data to MG-RAST, we were unable to find 34.0%, meaning that one-third of 
the reported accession numbers for MG-RAST did not lead to sequence data (29 out of 85 arti¬ 
cles). This happened only for 6.2% of the NCBI/ENA/DDBJ accession numbers (difference: 
Chisqi = 76.7, p < 0.0001). Furthermore, sequencing data was the only data type analysed 
here, and the archiving of other types of primary data is much less standardised and common 
compared to nucleic acid data [12,13]. For phylogenies, for example, it has been estimated that 
60% of the data are not available to science [14]. Guidelines for sequence submission are stan¬ 
dard for most journals, but such standards might not be true for other primary data; an analy¬ 
sis of the guidelines for authors of biomedical journals showed that only 12% of journals 
required general data sharing for publications [15]. Furthermore, the analysis presented here 
quantifies those studies that reported an accession number or link to sequence data, but it does 
not verify the quality of the attached metadata. Dubious metadata might additionally hamper 
the usefulness of submitted sequences for others (i.e., specific details of sampling conditions, 
location, associated environmental conditions, i.e., aerobic versus anaerobic, healthy or sick 
individuals, etc. [16]) and thereby increase the fraction of de facto unavailable metagenomes to 
science. 

Tax-payer-funded research must be open to the global research community. We acknowl¬ 
edge there are barriers to compliance and aU stakeholders need to ensure that these barriers 
are quickly and methodically addressed. Publishers without a history of publishing studies 
with metagenomic data need to learn the best practices from other publishers with a long and 
successful track record of ensuring Open Data. Data accessibility should be an ethical standard 
met by all researchers. 

Supporting information 

SI Text. Materials and methods. 

(DOCX) 

SI Data. List of journals with scientific fields. Fields, IF, publisher, and number of published 
papers according to Clarivate Analytics; DOAJ; main group according to the distinction 
between biological, multidisciplinary, or technological papers. DOAJ, Directory of Open 
Access Journals; IF, impact factor. 

(XLSX) 


PLOS Biology | https://doi.org/10.1371/journal.pbio.3000698 April 3, 2020 


5/7 




PLOS BIOLOGY 


S2 Data. The data set used for the analyses. It contains the 1,269 papers that passed the filter¬ 
ing steps of our literature survey on papers published between 2016 and 2019. 

(XLSX) 

51 Fig. Trend of accessibility of metagenomes over time. Data referrals for the year 2016 tiU 
March 2019. 

(TIFF) 

52 Fig. Difference in proportion of papers with accessible metagenomic data across scien¬ 
tific fields. Data collected according to Clarivate Web of Science. 

(TIFF) 

53 Fig. Differences between the proportion of data accessibility. Data related to the (A) pub¬ 
lisher, (B) journal impact factor, and (C) number of papers published. Trendlines in (B) and 
(C) are from the GLMs cited in Table 1. 

(TIFF) 

54 Fig. Graphical depiction of multicoUinearity between predictors used for statistical models. 

(TIFF) 
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